A technique for detecting the pupils of a subject can be used for detection of line of sight for medical testing or man-machine interface (see Patent Literature 1 below), drowsiness detection for a subject driving a vehicle (see Patent Literature 2 below), face orientation detection (see Patent Literature 3, 4 below), a pupil mouse for input to a computer or a game machine by the motion of the pupils of a subject without using a finger (see Patent Literature 5 below) and the like, for example. The present inventor has developed a technique to detect the pupils from images taken by a camera.
In the techniques disclosed in Patent Literatures 1 to 5, a method that distinguishes a pupil part from the surrounding image by subtraction between a light pupil image and a dark pupil image taken by a camera is used for detection of pupils. When placing a light source such as a near-infrared light source near the aperture of a camera and emitting light to the face of a subject along the light axis of the camera and taking an image, the light reaches the retina through the pupil and is reflected, then passes through the crystalline lens and the cornea and returns to the aperture of the camera. The pupil appears light in the image, and this image is referred to as a light pupil image. On the other hand, when emitting light from a light source away from the aperture of a camera to the face of a subject and taking an image, the light reflected from the retina does not substantially enter the aperture of the camera, and therefore the pupil appears dark in the image, and this image is referred to as a dark pupil image. The size of the pupil changes depending on the lightness of the surrounding, and it becomes small in a light place, which makes it difficult to be detected. Further, when the subject wears glasses, reflections occur in a part of the glasses near the pupil, and the glasses reflection appears in the image, and therefore it is difficult to detect the pupil from a single image even with use of any of the light pupil image and the dark pupil image. However, if subtraction of the dark pupil image from the light pupil image is performed, because the surrounding part other than the pupil part has almost the same lightness between both images, they cancel out each other, and only the pupil part having a difference in lightness is clearly shown. It is thereby possible to easily detect the pupil.
However, when using a common camera, there is a difference between the time to acquire a light pupil image and the time to acquire a dark pupil image. Therefore, when a face moves quickly during acquisition of the two images, a displacement occurs between the positions of the light pupil image and the dark pupil image, and the effect of image subtraction cannot be sufficiently obtained. Particularly, such a phenomenon is significant when a subject wears glasses, and glasses reflection, which is an image of light that is reflected on the lens or frame of the glasses, is likely to remain after the subtraction of the images, and the remaining glasses reflection image and the pupils cannot be easily distinguishable in some cases.
Thus, in order to prevent misdetection of a pupil caused by the above-described glasses reflection, a technique is introduced that, after detecting a pupil, applies an analysis window to a small region including the detected pupil in the image taken after that, and then tracks the pupil by searching for the pupil only in the small region, thereby reducing the pupil misdetection rate (for example, see Patent Literature 6).
Further, as a measure against the problem that the position of the pupil is displaced due to a difference in acquisition time of a light pupil image and a dark pupil image and it becomes difficult to detect position of the pupil, a subtraction position correction method using the amount of movement of nostrils (for example, see Patent Literature 7) and a subtraction position correction method using the amount of movement of corneal reflection (for example, see Patent Literature 8) are proposed, and their effectiveness is shown.
Furthermore, in the subtraction position correction method using the amount of movement of nostrils, when the head of a subject rotates around the axis penetrating the subject's head front to rear, the amount of movement of nostrils and the amount of movement of the pupils are not always the same, and the pupil detection accuracy is not high enough in some cases. In view of this, the present inventor has proposed a subtraction position correction method using a facial posture (for example, see Patent Literature 9).